Tiered cloud storage for different availability and performance requirements

ABSTRACT

A system and method for a tiered cloud storage for different availability and performance requirements includes a gateway, a block store configured to cache data, and an object store configured to persistently store data. The gateway, the block store, and the object store are in a compute zone. The gateway may receive from a user application a file access call and process the file access call. The gateway may also send the file access call to the block store. Then, the gateway may determine to store data in the object store and flush the data from the block store to the object store.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of application Ser. No.15/240,645 filed on Aug. 18, 2016, the entire content of which is herebyincorporated by reference herein.

BACKGROUND

Cloud computing defines a model for enabling ubiquitous, convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, and services)that can be rapidly provisioned and released with minimal managementeffort or service provider interaction. Essential characteristics of thecloud computing model include on-demand self-service, broad networkaccess, resource pooling, rapid elasticity, and measured service. Thecloud computing model includes several service models, includingSoftware as a Service (SaaS), Platform as a Service (PaaS), andInfrastructure as a Service (IaaS). The cloud computing model may beimplemented according to one of the several deployment models, includingprivate cloud, community cloud, public cloud, and hybrid cloud.

Cloud infrastructure is a collection of hardware and software thatimplements the cloud computing model. Cloud infrastructure may be viewedas including a physical layer and an abstraction layer. The physicallayer may include hardware resources designed to support the cloudservices being provided, and typically includes server, storage, andnetwork components. The abstraction layer may include the softwaredeployed across the physical layer, which manifests the essential cloudcharacteristics. Conceptually, the abstraction layer resides above thephysical layer. One type of IaaS is cloud storage. Cloud storage is adata storage service that provides storage to users in the form of avirtualized storage device over a network. Using cloud storage service,users may store, retrieve, maintain, and back up data remotely.

SUMMARY

The present disclosure provides new and innovative methods and systemsfor a tiered cloud storage. An example method includes receiving, by afirst gateway in a first compute zone, from a user application a fileaccess call and processing, by the first gateway, the file access call.The method also includes sending, by the first gateway, the file accesscall to a first block store that is configured to cache data. The methodfurther includes determining, by the first gateway, to store first datain a first object store that is configured to persistently store dataand flush the first data from the first block store to the first objectstore.

An example system includes a first gateway, a first block storeconfigured to cache data, and a first object store configured topersistently store data. The first gateway, the first block store, andthe first object store may be in a first compute zone. The first gatewaymay receive from a user application a file access call and process thefile access call. The first gateway may also send the file access callto the first block store and determine to store first data in the firstobject store and flush the first data from the first block store to thefirst object store.

Additional features and advantages of the disclosed methods and systemare described in, and will be apparent from, the following DetailedDescription and the Figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of an example tiered cloud storage systemaccording to an example of the present disclosure.

FIG. 2 a flowchart illustrating an example method of determiningfrequencies of the replication of data cache and metadata.

FIG. 3 is a flowchart illustrating an example method of operating atiered cloud storage according to an example of the present disclosure.

FIGS. 4A and 4B are flow diagrams illustrating example methods ofoperating a tiered cloud storage according to examples of the presentdisclosure.

FIG. 5 is a block diagram of an example tiered cloud storage systemaccording to an example of the present disclosure.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Described herein are methods and systems for a tiered cloud storage fordifferent availability and performance requirements. Object store is atype of data storage where data is stored in objects. Generally, objectsare stored in a flat structure, where objects are not organized in ahierarchy and one object is not stored in another object. Typically,each object includes the data to be stored in the object, variousmetadata, and a globally unique identifier within the object storagesystem. Object storage is cheaper, but slower than block storage. Blockstorage in cloud services emulates the storage of a traditional blockdevice, such as a physical hard drive, using a file system applied ontop of the block storage to map data onto blocks. Due to the performanceand consistency limitations, existing and legacy enterprise applicationsusually do not use object store. Existing solutions that emulate objectstore to file store generally do not perform well and have lowscalability.

Aspects of the present disclosure may address the above noted deficiencyby providing a tiered cloud storage for different availability andperformance requirements. Some cloud storage providers have providedtiered cloud storage services and cloud users may choose the appropriatestorage tier based on capacity, speed, and price. Tiered storage refersto the assignment of different types of data to different types ofstorage media. Aspects of the present disclosure may provide new andinnovative methods and systems for a tiered cloud storage based on adifferent data consistency model. In an example, a tiered cloud storagesystem may combine both fast yet expensive block store and slower butless expensive object store. The system may include a gateway (or acontroller), a block store, and an object store in a compute (or cloud)zone. When the gateway receives a write access call for data from a userapplication, the gateway may send the write access call to the blockstore and determine whether to store the data in block store and whetherto flush the data to the object store.

Metadata used by the gateway is replicated to other gateways acrossdifferent compute zones. Cached data blocks in the block store are alsoreplicated to other block stores across compute zones. Frequencies ofthe replication of the data cache and the metadata may be determinedheuristically. For example, if a cloud user prioritizes both dataconsistency and data loss rather than cost efficiency, both the metadataand data cache are replicated synchronously. If a cloud user prioritizesdata consistency rather than data loss, only the metadata is replicatedsynchronously, and the data cache is replicated asynchronously. If acloud user prioritizes cost efficiency rather than data consistency anddata loss, both the data cache and metadata are replicatedasynchronously. In this way, cloud users can choose the appropriatefrequencies based on price, data consistency, or data loss. Therefore,aspects of the present disclosure may offer a consistency and highavailability model that can be tailored to the needs of a particularuser and be highly scalable.

FIG. 1 depicts a high-level component diagram of an example tiered cloudstorage system 100 for different availability and performancerequirements. In an example, the system 100 may be a Portable OperatingSystem Interface (POSIX) compliant filesystem. The system 100 mayinclude one or more compute zones 110A-B. As used herein, a compute zonemay refer to a separate cloud region. The cloud region may be ageographical region where certain compute resources are located. Eachcompute zone 110A-B may include a gateway 120A-B, a block store 130A-B,and an object store 140A-B. In an example, the gateway may be a POSIXlayer. The gateway 120A-B may act as a bridge between user applicationsand remote cloud storage. The block store 130A-B may be a block storagedevice, which is a type of data storage where data is stored in blocks.The gateway 120A-B may be communicatively coupled to the block store130A-B. In an example, the block store 130A-B may be configured to cachedata. The gateway 120A-B may read or write cached data blocks in theblock store 130A-B. The block store 130A-B may be communicativelycoupled to the object store 140A-B. In an example, the object store140A-B may be configured to persistently store data. The block store130A-B may transfer data to or from the object store 140A-B. Forexample, data may be transferred to or from the object store 140A-B byexecuting “GET” or “PUT” operations.

The system 100 may include one or more customer sites 150A-B. As usedherein, customer sites 150A-B may refer to geographical regions. Eachcustomer site 150A-B may be associated with one or more compute zones110A-B. For example, the customer site 150A and the compute zone 110Amay be in the same geographic region A (e.g., in Boston), and thecustomer site 150B and the compute zone 110B may be in the samegeographic region B (e.g., in Chicago). Using a Master SelectionProtocol, one or more customer sites are selected as the primary siteand the remaining customer sites become backups. In an example, it maydepend on the location of the cloud user. For example, if a cloud user Ais located in region A, in which a customer site 150A and a compute zone110A are located, the customer site 150A is acting as primary to thecloud user A and the remaining customer sites (e.g., 150B) are acting asbackups. If a cloud user B is located in region B, in which a customersite 150B and a compute zone 110B are located, the customer site 150B isacting as primary to the cloud user B and the remaining customer sites(e.g., 150A) are acting as backups. Each customer site may include oneor more user applications 152A-B. Each customer site 150A-B may becommunicatively coupled to a gateway 120A-B in a compute zone 110A-Bassociated with the customer site 150A-B. In an example, cloud users mayneed to mount a file share to have access to the filesystem.

In an example, the customer sites may include a plurality of primarysites and a plurality of backup sites. In an example, only the primarysite(s) may be enabled to fully read and write to the block store andobject store in a compute zone associated with the primary site(s). Thebackup site(s) may be enabled to only read the block store and objectstore in a compute zone associated with the backup site(s).

In an example, metadata used by the gateway 120A may be replicated toother gateways 120B across different compute zones. In an example, themetadata may be information regarding which data is stored in the blockstore and which data is flushed to the object store and stored in theobject store. In another example, the metadata may include anyinformation related to the data stored in the filesystem, such asauthor, file size, date created, date last accessed, and date lastmodified. Cached data in the block store 130A may also be replicated toother block stores (e.g., 130B) across compute zones. Frequencies of thereplication of the data cache and the metadata may be determinedheuristically. Examples of the determination processes are described ingreater detail below and as shown in FIG. 2.

FIG. 2 shows a flowchart of an example method of determining frequenciesof the replication of data cache and metadata. Although the examplemethod 200 is described with reference to the flowchart illustrated inFIG. 2, it will be appreciated that many other methods of performing theacts associated with the method may be used. For example, the order ofsome of the blocks may be changed, certain blocks may be combined withother blocks, and some of the blocks described are optional.

In the illustrated example, the method 200 may include a step ofdetermining whether a cloud user prioritizes cost efficiency rather thandata consistency and data loss (block 202). For example, a useradministrator of a customer site 150A-B may set priorities and costsbased on system needs. If the user is more concerned about costefficient cloud storage service rather than data consistency and dataloss, both data cache and metadata may be replicated asynchronouslyacross the compute zones (block 204). For example, the data cache in theblock store 130A and the metadata in the gateway 120A may be replicatedto the block store 130B and the gateway 120B asynchronously. If the userdoes not prioritize cost efficiency over data consistency and data loss,the method 200 may include determining whether the user prioritizes dataconsistency rather than data loss (block 206). If the user is moreconcerned about data consistency rather than data loss, the metadata maybe replicated synchronously, but the data cache may be replicatedasynchronously across compute zones (block 208). For example, themetadata in the gateway 120A may be replicated to the gateway 120Bsynchronously, but the data cache in the block store 130A may bereplicated to the block store 130B asynchronously.

If the user does not prioritize data consistency over data loss, themethod 200 may include determining whether the cloud user prioritizesdata loss rather than data consistency (block 210). If the user is moreconcerned about data loss rather than data consistency, the cache datamay be replicated synchronously, but the metadata may be replicatedasynchronously across compute zones (block 212). For example, the datacache in the block store 130A may be replicated to the block store 130Bsynchronously, but the metadata in the gateway 120A may be replicated tothe gateway 120B asynchronously. If the cloud user prioritizes both dataconsistency and data loss, both the metadata and the data cache may bereplicated synchronously (block 214). For example, the data cache in theblock store 130A and the metadata in the gateway 120A may be replicatedto the block store 130B and the gateway 120B synchronously.

As used herein, replicating data synchronously may involve replicatingdata at substantially the same time that the data is created or updated.As used herein, replicating data asynchronously may involve notreplicating the data at the same time that the data is created orupdated. In this case, data may be replicated periodically. For example,the data may be replicated every minute, every 5 minutes, every 10minutes, or every hour.

FIG. 3 shows a flowchart of an example method of operating a tieredcloud storage according to an example of the present disclosure.Although the example method 300 is described with reference to theflowchart illustrated in FIG. 3, it will be appreciated that many othermethods of performing the acts associated with the method may be used.For example, the order of some of the blocks may be changed, certainblocks may be combined with other blocks, and some of the blocksdescribed are optional.

In the illustrated example, a gateway may receive, from a userapplication, a file access call (block 305). For example, the gateway120A may receive, from a user application 152A, a file access call.Examples of the file access call may include a write access call and aread access call. The gateway may process the file access call (block310). In an example, the gateway 120A may map the file access call toblock access. Then, the gateway may send the file access call to a blockstore that is configured to cache data (block 315). For example, thegateway 120A may send the file access call to a block store 130A. Then,the gateway may determine to store first data in an object store that isconfigured to persistently store data and flush the first data from theblock store to the object store (block 320). For example, the fileaccess call received from the user application 152A was a write accesscall to store first data, and the gateway 120A may determine to storethe first data in an object store 140A that is configured topersistently store data and flush the first data from the block store130A to the object store 140A.

FIG. 4 shows a flowchart of an example method 400 for a tiered cloudstorage according to an example of the present disclosure. Although theexample method 400 is described with reference to the flowchartillustrated in FIG. 4, it will be appreciated that many other methods ofperforming the acts associated with the method may be used. For example,the order of some of the blocks may be changed, certain blocks may becombined with other blocks, and some of the blocks described areoptional.

In the illustrated example, a user application may send a write accesscall for first data to a gateway (block 402). For example, the userapplication 152A may send a write access call to store first data to thegateway 120A. The gateway may receive the write access call (block 404).For example, the write access call may be writing text in a document inuser application 152A. Then, the gateway may process the write accesscall and map the write access call to block access (block 406). In anexample, the gateway 120A may determine whether to store the first dataonly in a block store 130A or whether to flush the first data to anobject store 140A and store it in the object store 140A. Then, thegateway may send the write access call to a block store (block 408) andthe block store may receive the write access call (block 410). Forexample, the gateway 120A may send the write access call to a blockstore 130A and the block store 130A may receive the write access call.The block store 130A may be configured to cache data and act as datalanding zone. In an example, the gateway may determine to store thefirst data in an object store and flush the first data to the objectstore (block 412). For example, the gateway 120A may determine to storethe first data in an object store 140A and flush the first data to theobject store 140A. The object store 140A may be configured topersistently store data. The block store may store the first data (block414) and flush the first data to the object store (block 416). Forexample, the block store 130A may store the first data and flush thefirst data to the object store 140A. In another example, the gateway120A may determine to store the first data only in the block store 130A.Then, the block store 130A may store the first data and the first datamay not be flushed to the object store 140A.

In an example, a user application may send a read access call (block418) and a gateway may receive the read access call (block 420). Forexample, the user application 152A may send a read access call for dataand the gateway 120A may receive the read access call. For example, theread access call may be reading texts or documents stored in thefilesystem such that the texts or documents can be retrieved and servedto the user application 152A. In an example, the gateway may checkwhether the data is stored in the block store or whether it is stored inthe object store. If it is determined that it is stored in the objectstore, the gateway may determine whether an object for the data in theobject store is modified (block 422). For example, the gateway 120A maydetermine whether the object for which the read access call was sent ismodified. If the object was modified, the gateway may determine whetherit is safe to retrieve the object (block 424). If it is safe, thegateway may retrieve the object (block 426). A block associated with theobject may be retrieved from the block store (block 428). In an example,the modification to the object may be updated to the block stores or thegateways across the compute zones. The object may be retrieved from theobject store (block 430). Then, the object may be retrieved and servedto the user application (block 432). In an example, the gateway maydetermine that it is safe to retrieve the object if the metadata orcache data related to the object have been updated and/or replicated toother gateways and block stores across the compute zones. If it is notsafe, the gateway may determine not to retrieve the object. In anexample, the gateway may delay the retrieval of the object until itbecomes safe (e.g., the modification is updated to block stores orgateways across the compute zones).

In an example, a user application may send a second write access callfor second data (block 434). For example, the user application 152A maysend a second write access call to store second data. For example, thesecond write access call may be writing text in a second document inuser application 152A. Then, the gateway may determine that the blockstore exceeds a predetermined threshold capacity (block 436). Forexample, the predetermined threshold capacity may be eighty (80), ninety(90), or ninety-five (95) percent of the full capacity of the blockstore. In another example, the predetermined threshold capacity may bedetermined based on the size of the remaining free storage area. Forexample, if the remaining storage area is less than 100 MB, 500 MB, 1GB, or 10 GB, the gateway determines that the predetermined thresholdcapacity is exceeded. In an example, the gateway may throttle the speedof processing the second write access call (block 438). For example, thegateway 120A may delay processing the second write access call for 1second to allow other data to be flushed from the block store 130A tothe object store 140A. If the threshold capacity is still exceeded, thegateway may expand the capacity of the block store (block 440) and theblock store capacity may be expanded (block 442). In an example, thegateway may expand the block store capacity without throttling the speedof processing the second write access call. In an example, additionalblocks may be added at a predetermined cost for a predetermined timeperiod.

FIG. 5 shows a block diagram of an example tiered cloud storage systemaccording to an example of the present disclosure. As illustrated inFIG. 5, an example system 500 may include a gateway 520, a block store530, and an object store 540. The block store 530 may be configured tocache data. The object store 540 may be configured to persistently storedata. The gateway 520, the block store 530, and the object store 540 maybe in a compute zone 510. The system 500 may also include a userapplication 552. The gateway 520 may receive from the user application552 a file access call 560. In an example, the file access call may be awrite access call for first data 570. The gateway 520 may process thefile access call 560 and send the file access call 560 to the blockstore 530. Then, the gateway 520 may determine to store the first data570 in the object store 540 and flush the first data 570 from the blockstore 530 to the object store 540.

It will be appreciated that all of the disclosed methods and proceduresdescribed herein can be implemented using one or more computer programsor components. These components may be provided as a series of computerinstructions on any conventional computer readable medium or machinereadable medium, including volatile or non-volatile memory, such as RAM,ROM, flash memory, magnetic or optical disks, optical memory, or otherstorage media. The instructions may be provided as software or firmware,and/or may be implemented in whole or in part in hardware componentssuch as ASICs, FPGAs, DSPs or any other similar devices. Theinstructions may be configured to be executed by one or more processors,which when executing the series of computer instructions, performs orfacilitates the performance of all or part of the disclosed methods andprocedures.

The examples may be embodied in the form of computer-implementedprocesses and apparatuses for practicing those processes. An example mayalso be embodied in the form of a computer program code containinginstructions embodied in tangible media, such as floppy diskettes,CD-ROMs, DVD-ROMs, hard drives, or any other computer readablenon-transitory storage medium, wherein, when the computer program codeis loaded into and executed by a computer, the computer becomes anapparatus for carrying out the method. An example may also be embodiedin the form of computer program code, for example, whether stored in astorage medium, loaded into and/or executed by a computer, ortransmitted over some transmission medium, such as over electricalwiring or cabling, through fiber optics, or via electromagneticradiation, where when the computer program code is loaded into andexecuted by a computer, the computer becomes an apparatus for carryingout the method. When implemented on a general-purpose microprocessor,the computer program code segments configure the microprocessor tocreate specific logic circuits.

It should be understood that various changes and modifications to theexamples described herein will be apparent to those skilled in the art.Such changes and modifications can be made without departing from thespirit and scope of the present subject matter and without diminishingits intended advantages. It is therefore intended that such changes andmodifications be covered by the appended claims.

The invention is claimed as follows:
 1. A method comprising: receiving,by a first gateway in a first compute zone, from a user application afile access call, wherein the first compute zone further includes afirst block store and a first object store, wherein the file access callis a write access call to store first data; determining, by the firstgateway, that the first object store exceeds a predetermined thresholdcapacity; responsive to determining that the first block store exceedsthe predetermined threshold capacity, throttling, by the first gateway,a speed at which the first gateway processes the file access call toallow other data to be flushed from the first block store to the firstobject store, wherein the first object store has a flat andnon-hierarchical structure, replicating metadata used by the firstgateway to a second gateway in a second compute zone, the metadatacomprising information regarding which data is stored in the first blockstore and which data is flushed to the first object store and stored inthe first object store, wherein the second compute zone further includesa second block store and a second object store, and replicating a datacache in the first block store to the second block store, wherein thedata cache stores data cached in the first block store.
 2. The method ofclaim 1, further comprising expanding, by the first gateway, thecapacity of the first block store.
 3. The method of claim 1, whereinfrequencies of the replication of the data cache and the metadata aredetermined heuristically.
 4. The method of claim 1, wherein both thedata cache and the metadata are replicated synchronously.
 5. The methodof claim 1, wherein only the metadata is replicated synchronously andthe data cache is replicated asynchronously.
 6. The method of claim 1,wherein only the data cache is replicated synchronously and the metadataare replicated asynchronously.
 7. The method of claim 1, wherein boththe data cache and the metadata are replicated asynchronously.
 8. Themethod of claim 1, further comprising: determining, by the firstgateway, whether an object in the first object store is modified,wherein the object is associated with a block in the first block store;and responsive to determining that the object is modified, determining,by the first gateway, whether it is safe to retrieve the object from thefirst object store.
 9. The method of claim 8, further comprising:responsive to determining that it is not safe to retrieve the objectfrom the first object store, delaying, by the first gateway, theretrieval of the object.
 10. The method of claim 9, wherein theretrieval of the object is delayed until the modification of the objectis updated across compute zones.
 11. A cloud storage system comprising:a first gateway; a first block store; and a first object store, whereinthe first gateway, the first block store, and the first object store arein a first compute zone, wherein the first gateway is configured to:receive from a user application a file access call, wherein the fileaccess call is a write access call to store first data; determine thatthe first object store exceeds a predetermined threshold capacity; andresponsive to determining that the first block store exceeds thepredetermined threshold capacity, throttle a speed at which the firstgateway processes the file access call to allow other data to be flushedfrom the first block store to the first object store, wherein the firstobject store has a flat and non-hierarchical structure, wherein metadataused by the first gateway is replicated to a second gateway in a secondcompute zone, the metadata comprising information regarding which datais stored in the first block store and which data is flushed to thefirst object store and stored in the first object store, wherein thesecond compute zone further includes a second block store and a secondobject store, and wherein a data cache in the first block store isreplicated to the second block store, wherein the data cache stores datacached in the first block store.
 12. The cloud storage system of claim11, wherein the first gateway is further configured to expand thecapacity of the first block store.
 13. The cloud storage system of claim11, wherein both the data cache and the metadata are replicatedsynchronously.
 14. The cloud storage system of claim 11, wherein onlythe metadata is replicated synchronously and the data cache isreplicated asynchronously.
 15. The cloud storage system of claim 11,wherein only the data cache is replicated synchronously and the metadataare replicated asynchronously.
 16. The cloud storage system of claim 11,wherein both the data cache and the metadata are replicatedasynchronously.
 17. The cloud storage system of claim 11, wherein thefirst gateway is further configured to: determine whether an object inthe first object store is modified, wherein the object is associatedwith a block in the first block store; and responsive to determiningthat the object is modified, determine whether it is safe to retrievethe object from the first object store.
 18. The cloud storage system ofclaim 17, wherein the first gateway is further configured to: responsiveto determining that it is not safe to retrieve the object from the firstobject store, delay the retrieval of the object.
 19. The cloud storagesystem of claim 18, wherein the retrieval of the object is delayed untilthe modification of the object is updated across compute zones.
 20. Acomputer readable medium including instructions that, when executed byone or more processors in a cloud storage system, cause the cloudstorage system to: receive, by a first gateway in a first compute zoneof the cloud storage system, from a user application a file access call,wherein the first compute zone further includes a first block store anda first object store, wherein the file access call is a write accesscall to store first data; determine, by the first gateway, that thefirst object store exceeds a predetermined threshold capacity; andresponsive to determining that the first block store exceeds thepredetermined threshold capacity, throttle, by the first gateway, aspeed at which the first gateway processes the file access call to allowother data to be flushed from the first block store to the first objectstore, wherein the first object store has a flat and non-hierarchicalstructure, wherein metadata used by the first gateway is replicated to asecond gateway in a second compute zone, the metadata comprisinginformation regarding which data is stored in the first block store andwhich data is flushed to the first object store and stored in the firstobject store, wherein the second compute zone further includes a secondblock store and a second object store, and wherein a data cache in thefirst block store is replicated to the second block store, wherein thedata cache stores data cached in the first block store.